Papers with computational savings
Efficient LLM Comparative Assessment: A Product of Experts Framework for Pairwise Comparisons (2024.emnlp-main)
Copied to clipboard
| Challenge: | LLM-as-a-judge approaches are effective but cost scales quadratically with number of candidates, which has practical limitations. |
| Approach: | They propose a Product of Expert (PoE) framework for efficient LLM Comparative Assessment where individual comparisons are considered experts that provide information on a pair’s score difference. |
| Outcome: | The proposed framework can generate score predictions that correlate well with human judgements on multiple NLG tasks with as few as 2% of comparisons. |
RAISE: Reinforced Adaptive Instruction Selection For Large Language Models (2025.findings-emnlp)
Copied to clipboard
Qingsong Lv, Yangning Li, Zihua Lan, Zishan Xu, Jiwei Tang, Tingwei Lu, Yinghui Li, Wenhao Jiang, Hong-Gee Kim, Hai-Tao Zheng, Philip S. Yu
| Challenge: | Existing selection methods rely on static, heuristic quality scores and are executed only once before training. |
| Approach: | They propose a dynamic selection framework that integrates selection into every training step. |
| Outcome: | The proposed framework integrates selection into every training step. |
Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking (2023.findings-acl)
Copied to clipboard
| Challenge: | Using a Bayesian optimization framework, we pre-train Transformer-based language models (TLMs) using a multi-armed bandit framework requires high computational resources and introduces many unresolved design choices. |
| Approach: | They propose a Bayesian optimization framework for resource efficient pre-training of Transformer-based language models. |
| Outcome: | The proposed framework achieves lower MLM loss in fewer epochs, across settings, while avoiding expensive hyperparameter grid search. |
Structured Pruning for Diverse Best-of-N Reasoning Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on the MATH dataset demonstrate that our method significantly outperforms traditional best-of-N and random head selection strategies. |
| Approach: | They propose a contrastive learning framework that dynamically selects the optimal head and layer to prune during inference by aligning question embeddings with head embedds. |
| Outcome: | The proposed approach outperforms best-of-N and random head selection strategies on the MATH500 and GSM8K datasets. |